In the practical application scenario, the wheat kernel image background is single, and the imperfect characteristics of wheat imperfect grains are mostly local features while most of the image features are not different from normal grains. In order to solve the problems, an imperfect wheat kernel recognition method based on detail Image Enhancement (IE) was proposed. Firstly, the alternate minimization algorithm was used to constrain the L0 norms of the original image in the horizontal and vertical directions to smooth the original image as the base layer, and the original image was subtracted from the base layer to obtain the detail layer of the image. Then, the detail layer was delighted and superimposed with the base layer to enhance the image. Finally, the enhanced image was used as the input of the Convolutional Neural Network (CNN), and the CNN with Batch Normalization (BN) layer was used for recognition of the image. The classic classification networks LeNet-5, ResNet-34, VGG-16 and these networks with the BN layer were used as classification networks, and the images before and after enhancement were used as input to carry out classification experiments, and the accuracy of the test set was used to evaluate the performance. Experimental results show that by adding the BN layer and using the same input, all three classic classification networks have the accuracy of the test set increased by 5 percentage points, and when using the images with enhanced detail as input, the three networks have the accuracy of the test set increased by 1 percentage point, and when the above two are used together, all the three networks obtain the accuracy of the test set improved by more than 7 percentage points.